In [18]:
__nbid__ = '0040'
__author__ = 'Brian Merino <brian.merino@noirlab.edu>, Susan Ridgway <susan.ridgway@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd; version datestamp of this notebook
__keywords__ = ['gmos','gemini','galaxy','dragons']

Gemini GMOS galaxy photometry data reduction using DRAGONS Python API¶

Public archival data from program GN-2017A-SV-151¶

adapted from https://dragons.readthedocs.io/projects/gmosimg-drtutorial/en/v3.0.3/03_api_reduction.html¶

Table of contents¶

  • Goals
  • Summary
  • Disclaimers and attribution
  • Imports and setup
  • About the dataset
  • Downloading data for reduction
  • Set up the DRAGONS logger
  • Create File Lists
  • Create Master Bias
  • Create Master Flat Field
  • Create Master Fringe Frame
  • Reduce Science Images
  • Display stacked final image
  • Clean-up (optional)

Goals¶

Showcase how to reduce GMOS imaging data using the Gemini DRAGONS package on the Data Lab science platform, using a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing flats, bias, and science frames, and creating a single combined stacked image.

Summary¶

DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It can currently be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, as well as spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals, and tutorials about the use of DRAGONS.

The DRAGONS kernel has been made available in the Data Lab environment, allowing users to access the routines without being dependent on installing the software on their local machines.

In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to reduce example Gemini South GMOS G-band imaging data fully. This notebook will not present all the details of the many options available to adjust or optimize the DRAGONS GMOS data reduction process; instead, it will just show one example of a standard reduction of a GMOS imaging dataset.

The data used in this notebook example is GMOS G band imaging from the Gemini archive of the galaxy NGC 5018 from the Gemini South program "The Evolutionary History of NGC 5018", PI: L. Sesto, program ID GS-2018A-Q-207. The final reduced science image combines five science frames of 460 seconds each that were dithered between each exposure.

Disclaimer & attribution¶

Disclaimers¶

Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.

Acknowledgments¶

If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:

This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:

This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

In either case please cite the following papers:

  • Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445

  • Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411

If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:

  • Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097

If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}

And if you are using SPARCL, please also add \software{SPARCL} and cite:

  • Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024 https://doi.org/10.48550/arXiv.2401.05576

The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.

For this notebook specifically, please acknowledge:

  • DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L

  • DRAGONS open source software publication

Importing Python libraries¶

In [3]:
import warnings
import glob

from gempy.adlibrary import dataselect
from gempy.utils import logutils

from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce

from astropy.io import fits
from astropy.wcs import WCS
from astropy.utils.exceptions import AstropyWarning

import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm

warnings.simplefilter('ignore', category=AstropyWarning)

About the dataset¶

The data used for this tutorial comes from a GMOS South program that observed the galaxy NGC 5018.

The table below contains a summary of the dataset downloaded in the previous section:

Observation Type File name(s) Purpose and Exposure (seconds)
Science S20180419S0098-102 460 s, g-band
Twilight Flats S20180419S0207-211 g-band
Bias S20180423S0050-054 0 s, r-band

Downloading the data¶

Downloading G-band images from the Gemini archive to the current working directory. This step only needs to be executed once.

If you run this notebook for the first time and need to download the dataset, set the variable "download=True". The notebook will not redownload the dataset if it is set to False. This will become particularly useful if you run the notebooks more than once.

In [4]:
%%bash 

# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/S20180419S0098.fits
http://archive.gemini.edu/file/S20180419S0099.fits
http://archive.gemini.edu/file/S20180419S0100.fits
http://archive.gemini.edu/file/S20180419S0101.fits
http://archive.gemini.edu/file/S20180419S0102.fits
http://archive.gemini.edu/file/S20180419S0207.fits
http://archive.gemini.edu/file/S20180419S0208.fits
http://archive.gemini.edu/file/S20180419S0209.fits
http://archive.gemini.edu/file/S20180419S0210.fits
http://archive.gemini.edu/file/S20180419S0211.fits
http://archive.gemini.edu/file/S20180423S0050.fits
http://archive.gemini.edu/file/S20180423S0051.fits
http://archive.gemini.edu/file/S20180423S0052.fits
http://archive.gemini.edu/file/S20180423S0053.fits
http://archive.gemini.edu/file/S20180423S0054.fits\
" > gmos_im.list
In [5]:
%%bash

download="True"

if [ $download == "True" ]; then
    wget --no-check-certificate -N -q -i gmos_im.list

else
    echo "Skipping download. To download the data set used in this notebook, set download=True."
fi

Setting up the DRAGONS logger¶

DRAGONS comes with a local calibration manager that uses the same calibration association rules as the Gemini Observatory Archive. This allows reduce to make requests to a local light-weight database for matching processed calibrations when needed to reduce a dataset.

This tells the system where to put the calibration database. This database will keep track of the processed calibrations we are going to send to it.

In [6]:
logutils.config(file_name='gmos_data_reduction.log')
caldb = cal_service.set_local_database()
caldb.init("w")
In [7]:
all_files = glob.glob('S2018*[0-9].fits')
all_files.sort()

Create file lists¶

The first step is to create input file lists. The tool "dataselect" helps with that. It uses Astrodata tags and "descriptors" to select the files and send the filenames to a text file that can then be fed to "reduce". (See the Astrodata User Manual for information about Astrodata.)

In [8]:
list_biases  = dataselect.select_data(all_files,['BIAS'],[])
list_flats   = dataselect.select_data(all_files,['FLAT'],[],
                                      dataselect.expr_parser('filter_name=="g"'))
list_science = dataselect.select_data(all_files,[],['CAL'],
                                      dataselect.expr_parser('(observation_class=="science" and filter_name=="g")'))

Create a master bias¶

We start the data reduction by creating a master bias for the science data. It can be created and added to the calibration database using the commands below. When the cell is done running, the master bias file will be saved with the name of the first bias and the suffix _bias.fits.

In [9]:
reduce_bias = Reduce()
reduce_bias.files.extend(list_biases)
reduce_bias.runr()

All submitted files appear valid:
S20180423S0050.fits ... S20180423S0054.fits, 5 files submitted.

Ginga not installed, use other viewer, or no viewer
No description has been provided for this image No description has been provided for this image

Create a master flat field¶

Twilight flat images are used to produce an imaging master flat and the result is added to the calibration database.

The master flat will have the name of the first twilight flat file with the suffix _flat.fits.

In [10]:
reduce_flats = Reduce()
reduce_flats.files.extend(list_flats)
reduce_flats.runr()

Create master fringe frame¶

The dataset used in this tutorial does not require a Master Fringe Frame. However, the following cell demonstrates how to create one for completeness.

The master fringe will have the name of the first science file with the suffix _fringe.fits.

In [11]:
reduce_fringe = Reduce()
reduce_fringe.files.extend(list_science)
reduce_fringe.recipename = 'makeProcessedFringe'
reduce_fringe.runr()

Reduce science images¶

Once our calibration files are processed and added to the database, we can run reduce on our science data.

This command will generate bias and flat-corrected files and will stack them. If a fringe frame is needed, this command will apply the correction. The stacked image will have the _stack suffix.

The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the "SCI" extension, the variance is in the "VAR" extension, and the data quality plane (mask) is in the "DQ" extension.

Each reduced science image will have the original name with the suffix _image.fits.

In [12]:
reduce_science = Reduce()
reduce_science.files.extend(list_science)
reduce_science.runr()

Display the stacked image¶

In [13]:
image_file = "S20180419S0098_image.fits"
hdu_list = fits.open(image_file)
wcs = WCS(hdu_list[1].header)
hdu_list.info()

Filename: S20180419S0098_image.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     250   ()      
  1  SCI           1 ImageHDU       148   (3245, 2182)   float32   
  2  VAR           1 ImageHDU       148   (3245, 2182)   float32   
  3  DQ            1 ImageHDU       148   (3245, 2182)   int16 (rescales to uint16)   
  4  PROVENANCE    1 BinTableHDU     17   7R x 4C   [28A, 128A, 128A, 128A]   
  5  PROVHISTORY    1 BinTableHDU     17   18R x 4C   [128A, 288A, 28A, 28A]
In [14]:
image_data = fits.getdata(image_file, ext=1)
print(image_data.shape)

(2182, 3245)
In [15]:
plt.figure(figsize = (15,15))
plt.subplot(projection=wcs)
plt.imshow(image_data,cmap='gray',norm=LogNorm(vmin=3000, vmax=60000),origin='lower')
plt.xlabel('Right Ascension [hh:mm:ss]',fontsize=14,fontweight='bold')
plt.ylabel('Declination [degree]',fontsize=14,fontweight='bold')
plt.xlim(900,2500)
plt.ylim(1600,600)
plt.show()
No description has been provided for this image

Optional: remove duplicate calibrations and remove raw data (uncomment lines before running)¶

In [17]:
# %%bash

# cp S20180419S0098_image.fits final.fits
# rm -r calibrations
# rm gmos_im.list gmos_data_reduction.log S2018*
# mv final.fits S20180419S0098_image.fits
In [ ]: